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Abstract 


The  battlefield  is  a  place  of  violence  ruled  by  uncertainty.  Timely  knowledge  of  what’s 
happening  around  a  soldier  can  mean  the  difference  between  life  and  death.  The  goals  of  an 
enhanced  mobile  infantry  are  becoming  a  reality  due,  in  part,  to  the  U.S.  Army’s  21st  Century 
Land  Warrior  (LW)  program.  However,  the  current  system  does  not  provide  a  “head  up”  display 
capability  like  that  provided  by  today’s  avionics.  When  the  soldier  employs  the  weapon,  he 
should  see  objects  easily  distinguishable  as  friendly  or  not,  as  well  as  enemy  locations.  The 
Eyekon  project  is  an  intelligent  agent-based  decision  support  system  hosted  on  the  soldier’s 
wearable  computer.  Eyekon  will  use  the  soldier’s  private  network  to  provide  a  perspective  view 
in  the  weapon  sight.  This  will  naturally  draw  the  warrior  to  the  most  desirable  target.  There  are 
many  performance  and  human  factors  issues  to  address  before  the  concept  can  be  used  in  lethal 
situations. 


If  you  load  a  mud  foot  down  with  a  lot  of  gadgets  that  he  has  to  watch, 
somebody  a  lot  more  simply  equipped  -  say  with  a  stone  ax  -  will  sneak  up  and 
bash  his  head  in  while  he  is  trying  to  read  a  vernier. " 

—Robert  Heinlein,  Starship  Troopers 


Introduction 

The  battlefield  is  a  place  of  violence,  chaos,  and  ruled  by  uncertainty.  Timely  knowledge  of  a 
soldier’s  environment  and  precise  team  coordinate  can  mean  the  difference  between  life  and 
death.  The  goals  of  an  enhanced  mobile  infantry  as  expressed  in  Heinlein’s  timeless  Starship 
Troopers  is  becoming  a  reality  due,  in  part,  to  progress  being  made  by  the  U.S.  Army’s  21st 
Century  Land  Warrior  (LW)  program.  However,  the  current  system  does  not  provide  a  “head  up” 
display  like  the  capability  that  today’s  avionics  provides  to  fighter  pilots.  What  is  desirable  is 
not  only  that  the  soldier  sees  objects  and  positions,  easily  distinguishable  as  friendly  or  not,  but 
also  the  means  to  designate  new  objects  as  possible  threats  and  to  coordinate  the  friendly  team’s 
actions  in  response  to  a  dynamically  changing  situation. 

This  work  is  aimed  at  developing  smart  icons  and  designations  that  are  superimposed  on  the  live 
video  from  a  soldier’s  weapon  sight.  21st  Century  Systems,  Inc.’s  Eyekon  is  an  intelligent 
agent-based  decision  support  system  hosted  on  the  soldier’s  wearable  computer.  Eyekon  will  use 
networked  position  reports  to  provide  a  perspective  view  in  the  weapon  sight.  The  system  will 
naturally  guide  the  warrior  to  the  most  desirable  target.  It  will  also  provide  a  means  for 
networked  coordination  of  fires  and  team  attacks  on  non-line-of-sight  targets.  Naturally,  there  are 
many  performance  and  human  factors  issues  to  address  before  the  concept  can  be  used  in  lethal 
situations. 

In  order  to  provide  the  dismounted  soldier  with  21st  Century  battlefield  dominance,  weapon 
systems  must  be  optimized  not  only  to  minimize  the  total  weight  carried  by  the  soldier  but  also 
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to  minimize  the  cognitive  overload.  Too  much  data  in  his  display  and  the  soldier  rapidly  is 
overwhelmed  with  “numbers,”  not  vital  situational  information.  Like  a  fighter  pilot’s  HUD  that 
displays  too  much  information,  the  battle  is  lost  before  it  starts.  Similarly,  too  much  information 
in  the  augmented  weapon  sight  and  the  soldier  becomes  a  battle  causality  due  to  a  missed  threat. 
What  if  a  soldier  could  point  out  an  imminent  threat  by  simply  marking  its  position  on  the 
display?  Then  all  networked  team-members  would  have  an  advance  warning  and  knowledge  that 
will  significantly  improve  the  chances  of  mission  success.  Further,  tactical  plans  can  be  adjusted 
to  the  present  situation  on  the  fly  and  coordinated  with  the  team  in  real-time,  all  without  making 
a  sound  or  a  move.  A  massed,  coordinated  attack  on  the  enemy  will  multiply  the  effectiveness  of 
a  soldier  many  times  over  through  the  power  of  information. 

These  weapon  systems  are  required  to  meet  the  tactical  battlefield  environmental  characteristics 
including  delivery  by  parachute  while  worn  by  the  Soldier.  They  must  be  self-contained,  man- 
packed,  and  battery-powered  since  maximum  efficiency  without  outside  help  is  a  must. 
Required  systems  must  not  rely  on  any  fixed  infrastructure  to  meet  the  operational  performance 
requirements. 

The  baseline  implementation  of  Eyekon  functions  on  a  wearable  computer  host  connected  via  a 
secure  wireless  network  to  local  and  remote  databases  and  to  real-time  sensor  streams  (e.g. 
inertial  attitude,  GPS,  IR).  Software  Agents  perform  network  queries,  monitor  threat  information, 
and  provide  in-situ  alerts  and  recommendations  to  the  soldier,  based  on  the  current  mission  and 
status  information.  The  computing  host  provides  a  simple,  yet  powerful  Augmented  Reality  (AR) 
overlay  on  weapon  sight  display  or  in  a  helmet-mounted  eyepiece.  The  HUD-like  display  utilizes 
size  and  brightness  for  perspective  and  color  and  shape  for  threat  information. 

The  Eyekon  built-in  decision  aids  provide  the  individual  soldier  more  in-depth  situational 
awareness  via  decision  graphics  (e.g.,  rated  targets  by  most  dangerous,  steering  arrow,  ROE 
issues,  team  status).  A  steering  arrow  points  to  next  priority  target,  if  outside  of  view,  then  the 
arrow  points  in  the  direction  of  the  acute  angle  and  the  arrow  length  is  proportional  to  angle 
magnitude.  Rules  of  Engagement  (ROE)  guidance  can  help  the  soldier  in  situations  where 
responses  are  escalating.  Team  status  provides  the  team’s  real-time  situation,  e.g.  rounds 
remaining,  water,  casualties.  An  extended  version  of  Eyekon  provides  support  at  the  team  level. 
Given  projected  computing  performance  capabilities,  processing  speed,  memory  capacity,  and 
network  bandwidth;  Eyekon  would  support  a  new  type  of  team  interaction:  self-organizing 
behavior.  The  Army  unit  with  Eyekon  divides  recon  and  target  tasks  as  a  self-organizing  team.  It 
is  very  robust  because  the  enemy  cannot  “take  out”  the  leader.  The  Team  is  very  adaptable  and 
responsive  to  changing  battlefield  conditions.  It  swarms  the  battlefield  and  instantly  responds  to 
the  unfolding  event,  in  part  due  to  the  capability  of  Eyekon  In-the-situation  awareness. 

The  applicability  of  the  Eyekon  technology  to  non-combatant  and  operations  other  than  war 
(OOW)  is  especially  attractive.  Eyekon’s  AR  interface,  combined  with  its  agent-based  decision 
aids,  has  the  potential  to  revolutionize  law  enforcement,  fire  and  disaster  management, 
maintenance  operations,  and  intelligence  gathering.  The  team-based  coordination  and 
(re)planning  functions  can  reduce  execution  time  and  increase  effectiveness,  thus  providing 
savings  in  terms  of  reduced  manning,  increased  productivity,  and  reduced  exposure  to  hazards. 
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Augmented  Reality 

Augmented  reality  (AR)  technology  provides  the  user  with  superimposed  information  that  can  be 
seen  in  the  real  world;  we  supplement  it.  Ideally,  it  would  seem  to  the  user  that  the  real  and 
virtual  objects  coexist.  Alignment  of  the  superimposed  information  is  critical.  Just  a  few  pixels 
off  and  the  “augmentation”  becomes  a  nuisance.  Add  latency  and  confusion  results  for  the  user. 
What  is  AR  good  for?  Basically,  applications  of  this  technology  use  the  virtual  objects  to  aid  the 
user's  understanding  of  his  environment.  AR  improves  the  perception  of  the  natural  world  by 
adding  to  our  senses  via  computer-generated  information  visual  graphics,  sounds,  smells  or 
tactical  sensations.  Televised  football  games  use  SporTVision’s  first-down  line  superimposed 
over  the  game  scene  to  increase  the  viewing  audience  awareness. 


Soldier  Systems 

Land  Warrior 


100ms 


milliseconds 


lms  lOOjus  lOjLis  1  jus 
microseconds 


Unix,  Solaris, 
HP-UX.  NT 


HV1 


Fig.l.  Requirements  for  Soldier  Systems 


Eyekon  involves  the  design  of  a  user  interface  and  is  a  subset  of  AR  of  visual  enhancements 
applied  to  the  Land  Warrior  concept.  Our  concept  is  aimed  not  at  the  current  configuration  (LW 
System  1.0)  but  next  version  since  it  would  be  too  costly  to  add  a  requirement  into  the  current 
design  and  there  are  limitations  in  the  weapon  sight  capability.  Our  study  configuration  is 
evolving  to  meet  the  domain  requirements  (Figure  1).  The  computer  output  is  directed  to  either 
the  weapon  sight  or  helmet  mounted  eyepiece.  The  eyepiece  can  be  used  as  a  situation  map 
display  or  as  the  weapon  sight  when  the  weapon  is  held  off  shoulder;  say  when  the  weapon  is 
pointed  around  the  comer  of  a  building. 

The  decision  support  design  will  leverage  the  graphical  user  interface  development  methodology 
from  projects  involving  LISA,  USAF  and  USN  along  with  our  real-time  Assistive  Agent 
development  environment  (AEDGE™)  to  provide  real-time  software  for  a  “heads-up”  weapon 
aiming  application.  With  proper  Land  Warrior  visualization  and  host  environment  requirements 
for  specific  Soldier’s  needs,  appropriate  planning  for  software  development  for  the  Eyekon 
concept  and  prototype  will  commence.  Eyekon  concept  features  are: 

* Superimpose  situation  icons  on  a  dismounted  Soldier 's  weapon  sight 

* Improve  situation  awareness  by  rating  threats  in  a  perspective  view 
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* Maximize  responsiveness  and  minimize  weight  and  power  requirements 

The  technical  objective  to  investigate  the  applicability  of  software  components  that  overlay  smart 
icons,  named  Eyekon,  to  the  dismounted  soldier’s  individual  weapon  sight.  Eyekon  provides 
objects  for  the  graphical  user  interface  of  friendly  and  enemy  positions  in  the  weapon  sight  when 
the  dismounted  Soldier  employs  the  weapon.  Eyekon  is  an  intelligent  agent-based  decision 
support  system  hosted  on  a  wearable  computer  with  an  available  database  and  updates  via  radio 
links.  The  output  of  Eyekon  is  fed  directly  to  the  soldier’s  display.  Display  examples  are  a 
hands-free  device  (HFD),  weapon-aiming  devices  that  could  be  installed  on  the  existing  Ml 6 
rifle,  the  M4  carbine,  the  future  Objective  Individual  Combat  Weapon  (OICW),  or  helmet- 
mounted  display  (HMD).  Eyekon  would  support  not  only  weapon  aiming  but  also  would  display 
combat  identification  of  friendly  soldier’s  response  to  interrogation  devices,  help  direct  fires  by 
highlighting  target  areas,  or  augment  the  direct  view  using  a  see-through  HMD.  The  developed 
software  components  must  conform  to  the  Joint  Technical  Architecture- Army.  This  is  a  vital 
requirement  for  enhancing  concepts  such  as  interoperability,  cost  avoidance,  and  force  multiplier 
due  to  fielded  asset  commonality. 

The  Eyekon  concept  is  comprised  of  a  baseline  set  of  display  symbols  and  an  optional  set  of 
functions  that  enhance  the  display.  The  use  of  the  options  depends  the  reported  usefulness 
through  customer  feedback.  The  proposed  implementation  minimizes  cognitive  workload  and 
requires  minimal  training. 

The  baseline  implementation  of  Eyekon  functions  using  a  wearable  computer  host  and  using 
existing  data,  network  queries,  and  computing  host  provide  perspective -view  icons  overlaid  on 
weapon  sight  display.  We  will  use  size  and  brightness  for  perspective.  The  decision  aid 
provides  the  individual  soldier  a  more  in-depth  situational  awareness  with  decision  graphics 
(e.g.,  rated  targets  by  most  dangerous,  steering  arrow,  ROE  issues,  team  status).  Steering  arrow 
points  to  next  priority  target  or,  if  outside  of  view,  then  the  arrow  points  in  direction  of  acute 
angle  and  the  arrow  length  is  proportional  to  angle  magnitude.  Rules  of  Engagement  (ROE)  can 
help  soldier  in  situations  where  escalation  responses  are  changing.  Team  status  provides  what 
the  team’s  real-time  situation  (e.g.,  rounds  remaining,  water,  casualties).  An  extended  version  is 
to  provide  support  at  the  team  level.  Given  slated  computing  performance  capabilities 
(processing  speed,  memory  capacity,  and  network  bandwidth),  Eyekon  would  support  a  new  type 
of  team  interaction:  self-organizing  behavior.  See  for  example,  the  article  from  US  Army  War 
College  [3],  The  Team  with  the  slated  field  capability  and  Eyekon  Option  2  divides  recon  and 
target  areas  as  a  self-organizing  team.  It  is  very  robust  because  the  enemy  cannot  “take  out”  the 
leader.  The  Team  is  very  adapt-able  and  responsive  to  changing  battlefield  conditions.  It 
swarms  the  battlefield  and  instantly  responds  to  the  unfolding  events,  in  part  due  to  the  capability 
of  Eyekon  In-the-situation  awareness. 


Technical  Approach 

The  technical  approach  is  to  investigate  and  to  analyze  dismounted  soldier  interface  requirements 
that  increase  situation  awareness  from  present  practice.  It  will  determine  a  set  of  system 
visualization  requirements  and  metrics,  to  perform  a  functional  analysis  and  develop  a  software 
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plan,  to  adapt  21st  Century  System’s  AEDGE™  software  and  to  host  it  on  a  soldier’s  wearable 
system.  The  “heads-up”  situation  awareness  is  first  defined  as  when  the  soldier  shoulders  the 
weapon  and  looks  though  the  weapon  aiming  sight;  a  HFD.  Alternate  “heads-up”  are:  (1)  being 
able  to  use  the  weapon  sight  off  the  weapon,  such  as  to  look  around  a  building  corner  without 
being  exposed,  (2)  a  helmet-mounted  display  (HMD)  consisting  an  eye-piece  that  fits  over  one 
eye  (not  see-through),  and  (3)  a  HMD  with  see-through  eyepiece.  User  interface  requirements 
and  related  resource  requirements  investigation  areas  that  will  be  addressed,  for  day,  twilight  or 
night  operations  are:  (1)  Heading  of  the  using  Soldier,  (2)  Position  of  all  friendly  units  and 
Soldiers,  (3)  Enemy  threat  locations,  if  available  by  Army  digital  Tactical  Internet,  (4)  Weapons 
orientation  of  friendly  Soldiers,  (5)  Identification  of  individual  Soldier  and  aggregate  units  and 
(6)  Ready  status  of  individual  Soldiers  in  terms  of  ammunition,  water  and  health. 

Data  requirements  via  situational  reports  fed  continuously  by  the  Soldiers  will  also  be 
investigated.  The  automated  reports  and  Eyekon  data  provide  the  situational  awareness  of  the 
team  to  a  Soldier  without  having  the  entire  unit  stop  and  take  a  formal  report  and  expose 
themselves  to  enemy  counter-attack  by  losing  momentum.  In  addition,  for  military  operations  in 
urban  terrain  (MOUT)  or  possibly  close-quarters  battle  (CQB)  where  non-combatants  are  in  the 
are  of  operations  (“no-shoot”  targets  mixed  with  known  threats),  the  study  will  investigate 
display  requirements  to  depict  friendly  locations  even  if  behind  a  wall  in  building  or  in  a  cave  by 
superimposing  an  Eyekon  over  their  relative  positions  through  the  weapons  sight. 

Another  requirement  is  that  Eyekon  must  perform  in  the  host  environment,  that  is,  be  rugged  to 
handle  the  warrior  mission.  Host  system  processing,  networking  and  data  storage  characteristics 
are  continuing  to  advance.  Wearable  system  technology  advances  continue  in  many  application 
areas  such  as  for  U.S.  Army  and  USSOCOM  projects.  Industries  always  want  worker’s 
productivity  to  increase.  Wearable  computers,  combined  with  wireless  technology,  allow  art 
dealers  to  examine  artifacts  on  location  and  instantly  compare  with  electronic  library  high- 
resolution  photographs.  Another  area  is  working  in  tight  quarters,  such  as  in  commercial 
aerospace  application  where  a  flight-line  technician  has  direct  access  to  all  technical  orders  while 
working  in  the  cramped  areas  (e.g.,  electronics  bay  of  the  aircraft).  The  point-of-departure 
Eyekon  concept  is  shown  in  Figure  2.  As  concept  matures,  the  baseline  is  expected  to  change. 
The  point-of-departure  provides  a  reference  with  which  to  compare  candidate  concepts.  Data  is 
provided  to  Eyekon  via  the  soldier’s  database.  Eyekon  provides  an  overlay  of  friendly  and 
enemy  icons  positioned  on  the  weapon  sight  video.  Position  accuracy  is  dependent  on  database 
data  accuracy.  Agents  maintain  Eyekon  updates  as  weapon  sight  is  moved  through  angles.  It  is 
comprised  of  a  baseline  set  and  an  optional  set  of  functions.  The  use  of  the  options  depends  the 
reported  usefulness  through  customer  feedback. 

A  point-of-departure  for  the  Eyekon  baseline  is  to  develop  symbols  using  data  from  the  soldier’s 
(e.g.  Land  Warrior)  database.  Determine  the  soldier  kinematics  such  as  present  position,  heading 
and  rate  of  change,  weapon  pointing  (compass,  tilt  and  rate  of  change)  and  possibly  head 
pointing  and  then  determine  kill  box  for  the  weapon’s  range  and  kinematic  state.  Agents  will 
query  database  and  tactical  network  for  target  position  reports  of  friendly,  enemy  &  neutral  in  or 
near  the  kill  box  and  develop  standard  object  display  icons  for  position  reports  and  track.  An 
intelligent  agent  will  adjust  the  standard  icons  for  perspective  view  objects  (smart  icons).  The 
perspective  view  is  based  on  identity  and  distance.  The  icon’s  display  attributes  such  as  color, 
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brightness  and  size  will  also  be  adjusted  for  time  of  day  to  account  for  vision  changes  due  to 
daylight,  twilight,  or  darkness.  It  will  also  adjust  weapon  sight  display  based  on  sight 
magnification,  heading,  tilt,  and  angular  rates.  It  will  register  for  weapon  sight  and  movement, 
and  then  display  icons. 


The  established  Land  Warrior  program  provides  guidance  for  assessment  (via  criteria  weighting) 
or  the  override  of  the  assessment  function.  The  assessment  determines  most  dangerous  threats 
based  upon  threat  vulnerability.  The  vulnerability  calculation  determines  LOS  distance  and  the 
threat’s  weapon  type  using  probability  of  hit  and  probability  of  kill  values  from  the  LW  database. 
It  adjusts  the  perspective  view  of  the  icons  based  on  assessed  threat  values,  ROE  and  current 
level  of  escalation.  It  will  provide  a  steering  arrow  that  will  point  to  the  next  priority  target(s). 
That  is,  if  next  target  is  inside  the  current  field  of  view  (E OV),  then  arrow  points  to  maximum 
threat  icon.  However,  if  next  target  is  outside  of  current  FOV,  then  arrow  points  in  shortest 
direction  (left  or  right)  and  the  arrow  length  is  proportional  to  angular  magnitude.  A  control 
deadband  would  remove  pointing  ambiguity  if  the  enemy  were  directly  behind.  Eyekon  decision 
aids  will  take  advantage  of  the  fact  that  the  team  is  networked  via  a  tactical  network.  The 
networked  team  is  technically  capable  of  being  a  team  that  is  self-organizing  and  non- 
hierarchical.  An  example  in  the  literature  is  New  York  City.  The  populace  is  feed  every  day. 
Yet,  no  one  is  in  charge  of  feeding  the  inhabitants.  No  one  organization  is  responsible  for 
ordering,  scheduling,  delivering,  or  expediting  food.  It  works  through  self-organizing 
commerce.  This  is  not  the  current  way  that  the  U.S.  Army,  or  any  army  or  any  national  military 


Fig.2.  Eyekon  Concept 


organization  works  today.  As  the  thrust  for  information  access  is  pushed  down  to  lower  and 
lower  echelon  levels,  ultimately  every  soldier  will  be  able  to  access  any  information  available  at 
any  time.  This  widespread  access  has  the  potential  for  new,  unheard-of  organization  schemes  to 
emerge.  See,  for  example,  The  Real  Military  Revolution  [3].  The  Team  with  this  capability  and 
Eyekon  Option  2  divide  up  recon  and  target  areas  as  a  self-organizing  team.  It  is  very  robust 
because  the  enemy  cannot  “take  out”  the  leader.  The  Team  is  very  adaptable  and  responsive  to 
changing  battlefield  conditions.  It  swarms  the  battlefield. 


The  concept  for  a  set  of  point-of  departure  (POD)  symbols  will  be  reviewed  with  subject  matter 
experts  and  trades  against  the  POD  set  will  be  made.  A  follow-on  study,  using  subject  matter 
experts,  will  consist  of  a  cognitive  task  analysis  (CTA)  performed  by  a  cognitive  psychologist. 
The  CTA  study  will  consider  individual  and  team  interactions.  Metrics  will  be  used  to  quantify 
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symbol  layout,  selections,  and  symbol  attributes.  Though  initial  work  is  focused  on  the 
individual  and  the  individual’s  interface,  team  interactions  will  become  more  important  as  use  of 
the  tactical  intranet  increases.  The  friendly  situation  is  shown  in  the  lower  right  portion  of  the 
weapon  sight. 

For  the  baseline,  a  requirements  analysis  will  develop  the  top-level  functions  for  the  Eyekon 
concept  and  be  presented  as  a  functional  flow  diagram.  The  functions  are  shown  in  Figure  3. 


Soldier  - 
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Weapon 
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Fig.3.  Eyekon  Functionality 


It  defines  four  functions.  The  first  function  is  “Locate  weapon”  and  sets  the  stage  for  the  display 
of  the  area  sector  -  just  where  is  the  soldier  in  the  area  and  where  is  the  weapon  (sight)  pointing 
and  how  is  it  changing.  As  depicted  at  the  top  in  the  figure  the  soldier,  interacting  with  Eyekon 
functions,  may  change  criteria  weightings  if  the  database  information  accuracy  changes  or  if  the 
weapon  that  the  sight  is  attached  changes.  Implementation  depends  upon  selection  of  options. 
With  the  soldier  and  weapon  position  now  known,  Eyekon  might  send  out  queries  on  the  LW 
network  for  situation  reports  within  the  weapon  range.  Periodic  report  updates  and  aperiodic 
queries  feed  Eyekon  object  evaluation.  Tactical  database  objects  are  identified  as  friend,  foe, 
neutral  or  unknown  and  located  within  the  FOV.  The  decision  aiding  then  scales  according  to 
leader  criteria  and  threat  level. 


Application  of  AEDGE™ 

We  take  a  pragmatic  approach  to  intelligent  agents,  as  we  apply  our  experience  delivering  agent 
technology  to  other  domains  (team  decision  support,  software  systems  synthesis).  Our  agents  are 
autonomously  running  software  components,  which  monitor,  analyze,  and  advise  human  users. 
Such  agents  will  add  great  utility  to  the  platoon.  Agents  provide  choices  and  evaluate  the 
performance  of  users  “on  the  fly”  to  give  valuable  feedback  to  the  participants.  We  propose  two 
classes  of  agents  in  our  tool  suite. 
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Agent  Types 

The  first  class  is  comprised  of  human-assistant  agents.  A  human  assistant  agent  provides 
concrete  advice  to  humans  who  are  participating  in  the  session.  Such  assistant  agents  could  be 
useful  both  in  the  learning  process  as  well  as  in  the  content  development  and  evaluation 
processes.  If  a  trainee  is  making  an  incorrect  choice  at  a  critical  point  in  the  exercise  on  which 
the  correctness  of  the  rest  is  dependent,  an  agent  can  provide  a  second  chance  or  additional  clues 
for  the  concrete  situation,  thus  highly  increasing  the  educational  value  for  the  trainee.  Human 
assistant  agents  both  react  to  human  requests  and  provide  advice  without  being  prompted  for  it, 
if,  they  determine  that  help  and  guidance  is  needed.  Human  assistant  agents  are  one  of  the 
contributions  of  this  work. 

Coordinators  and  arbitrators  of  distributed  decisions  populate  the  second  class  of  agents.  Even 
when  individual  agent  and  human  decisions  are  based  on  team-level,  global  knowledge  (which 
may  or  may  not  be  the  case,  depending  on  the  nature  of  a  decision  or  knowledge  available  to  a 
human  or  an  agent),  it  is  normal  to  expect  conflicts  and  also  temporal  discrepancies.  Agents  of 
the  third  kind  try  alleviating  such  conflicts,  advising  agents  and  humans  (including,  agents  of  the 
second  kind)  how  to  adjust  these  decisions  and  re-enter  a  more  stable  situation,  while  still  not 
sacrificing  their  (conflicting)  objectives.  Examples  of  conflicts  include  multiple  users  requesting 
modification  of  a  single  simulation  object  (e.g.  a  route  way  point  or  an  infiltration  zone)  at  the 
same  time,  or  a  user  attempting  to  switch  the  nature  or  the  pace  of  an  exercise  while  in  a  team 
mode  with  other  participants.  Agents  of  the  third  kind  too  are  a  major  contribution  of  this  work. 

The  need  for  a  standardized  common  infrastructure  has  lead  21CSI  to  design  an  environment 
where  both  agents  and  simulated  entities  (or  representations  of  real-world  assets)  are  represented 
as  first-class  objects  capable  of  interacting  with  each  other.  21CSI’s  extensible  multi-component 
Decision  Support  Systems  (DSS)  architecture,  known  as  AEDGE™,  is  a  standardized 
Commercial  Off  the  Shelf  (COTS),  DII  COE  compliant  agent  architecture  that  enables  complex 
DSS  to  be  developed  as  an  expansion  of  the  AEDGE™  core  functionality. 

The  Eyekon-AEDGE™  study  interface  is  shown  in  Figure  4.  The  developer  is  provided  a  set  of 
overlay  controls,  the  circle  “A”,  and  a  set  of  scenario  controls,  “B”.  The  picture  is  a  video 
capture  of  an  experimental  site.  The  overlay  controls  direct  either  video  feed  to  the  screen  or 
brings  up  a  2D  map  (simulating  the  warrior’s  HMD).  In  this  setup  five  people  are  positioned 
with  three  to  the  front  of  the  observer  with  a  video  camera  and  two  behind.  We  are  undertaking 
to  build  a  common  reference  framework  and  a  test-bed  environment  for  integrated  simulation 
and  agent-based  decision  support.  In  a  simulation  that  feeds  Eyekon  the  people  with  initial 
headings  begin  walking  and  the  observer  rotates  the  camera.  The  kernel  of  the  architecture 
consists  of  four  core  and  five  extender  components.  These  define  the  internal  structures, 
dataflow,  and  interface  to  the  architecture. 

AEDGE™  defines  Agents,  Entities,  Avatars  and  their  interactions  with  each  other  and  with 
external  sources  of  information.  This  standardized  architecture  allows  additional  components, 
such  as  service-specific  DSS  tools,  to  be  efficiently  built  upon  the  core  functionality.  Common 
interfaces  and  data  structures  can  be  exported  to  interested  parties  who  wish  to  extend  the 
architecture  with  new  components,  agents,  servers,  or  clients.  When  the  core  AEDGE™ 
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Fig.4.  Initial  Set  Up  with  Video  Only 

components  developed  by  21CSI  are  bundled  with  customer-specific  components  in  an 
integrated  environment,  a  clean  separation  of  those  components,  through  APIs,  is  provided. 
Distributed  components  communicate  via  Services.  Services  are  an  abstract  and  network-friendly 
representation  of  functions,  implemented  by  AEDGE™  components.  The  Service  object  is  a  data 
storage  and  transport  mechanism  for  a  service  request.  The  request  includes  the  class  and  the 
type  of  the  service,  an  optional  sub-type  and  a  list  of  parameters.  Return  values  for  services  are 
provided  via  ServiceResult  objects,  which  encapsulate  a  service-success  code  and  the  actual 
return  value. 

Shown  in  Figure  5,  the  developer  has  selected  the  video  with  Eyekon  overlay  during  a  simulation 
run.  Note  that  around  the  screen  is  information  on  weapon  position  (“C”  for  weapon  elevation, 
“D”  for  azimuth  or  compass  and  “F”  for  GPS  location).  In  the  screen  area,  friendly  positions  are 


Fig.5.  Video  with  Eyekon  Overlay 


©  Copyright  21st  Century  Systems,  Inc.  2001-2003 


10 


Eyekon:  Distributed  Augmented  Reality  for  Soldier  Teams 


noted  by  a  blue  box,  threats  by  a  red  diamond.  Tracks  developed  by  updated  position  reports  are 
given  an  identification  number  such  as  1301  for  the  friendly  and  1302  for  the  threat. 


Assisting  Situational  Awareness 

Maintaining  situational  awareness  both  in  a  training  and  in  an  operational  environment  is  the 
single  most  important  element  for  achieving  the  mission  objectives  as  well  as  for  improving  the 
training  process.  In  the  present  and  especially  in  the  future  reality  of  a  digital  battlefield,  the 
inflow  of  information  from  sensors,  intelligence  reports,  and  damage  assessment  most  certainly 
will  overwhelm  the  soldier  as  a  decision-maker  and  will  obscure  the  tactical  picture.  A  high  level 
of  situational  awareness  is  maintained  by  introducing  monitoring  and  advisory  agents.  The 
purpose  of  the  monitoring  agents  is  to  subscribe  to  specific  data-flows,  to  observe  and  analyze 
their  trends,  and  to  raise  flags  in  case  the  tactical  situation  has  changed  significantly.  Information 
fusion  techniques  will  be  applied  to  correlate  data  from  different  sources  in  order  to  increase  the 
certainty  of  the  tactical  picture.  Advisory  agents  occupy  the  next  level  in  the  information 
processing  hierarchy  -  they  subscribe  to  the  information  provided  by  the  monitoring  agents  and 
react  to  situational  changes  by  re-evaluating  the  threats  posed  to  our  forces  and  the  risks 
associated  with  the  mission. 


Fig.6.  Target  Selected 


Figure  6  is  a  result  of  the  decision  aiding  process  where  track  1302  has  been  selected  or 
“hooked”.  Hooked  refers  to  target  selected  from  either  the  map  or  sight  picture  and  tracked.  The 
“G”  points  to  the  target  (1302)  and  has  a  box  around  it  to  denote  that  special  attention  is  to  be 
given  to  this  object.  Object  is  identified,  in  lower  left  comer  of  the  screen,  “H”,  as  a  human  and 
as  a  hostile  target.  Advisory  agents  then  communicate  with  the  human  user  (via  specialized  user 
interfaces)  to  alert  him  of  the  changed  risks  and  the  underlying  reason  for  that  change,  such  as 
new  enemy  detected,  or  friendly  forces  that  have  suffered  excessive  damage.  Advisory  agents 
can  focus  the  user’s  attention  to  a  particular  area  of  the  tactical  picture  or  can  highlight  a  trend  in 
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the  picture  that  was  impossible  to  observe  due  to  information  overflow.  Eyekon  provides  two 
types  of  decision  aids:  1)  a  passive  decision  aid,  which  presents  the  right  information  at  the  right 
time  to  the  right  user;  2)  a  proactive  recommendation  agent,  which  provides  tactical  advice  and 
recommends  a  COA.  These  two  approaches  are  often  complementary  and  thus,  their  combined 
application  provides  greater  utility.  In  addition,  the  user  is  presented  with  a  flexible  decision 
support  environment,  which  adapts  to  the  current  tactical  requirements  as  well  as  to  his  personal 
preferences. 

The  passive  decision  aid  is  based  on  the  monitoring  and  advisory  agents  and  their  ability  to 
analyze  the  enormous  amounts  of  incoming  information  and  highlight  important  changes.  The 
agents  do  not  interact  directly  with  the  user,  but  rather  sit  in  the  background  and  monitor  the 
development  of  the  tactical  picture.  Once  the  picture  changes,  the  advisory  agents  dynamically 
reconfigure  the  user  display  to  show  these  changes  and  the  predictions  for  future  effects.  For 
example  the  monitoring  agents  may  detect  movement  of  the  enemy  forces  at  the  left  flank.  This 
movement  will  initially  trigger  re-evaluation  of  the  threat  based  on  the  enemy’s  new  positions 
and  simultaneously  will  invoke  a  heuristic  predictor  of  the  enemy’s  future  positions  and 
intentions. 

The  displayed  information  (enemy  movement  route  at  the  left  flank  and  future  positions)  will 
depend  on  the  likelihood  of  this  change  adversely  affecting  our  current  operations.  As  a  result  the 
new  threat  and  projected  future  threat  areas  will  be  shifted  toward  the  left  flank.  The  dynamic 
reconfiguration  based  on  heuristic  threat  estimation  goes  beyond  improving  the  situational 
awareness  of  the  user.  This  capability  provides  the  commander  with  clues  as  to  what  his  next 
action  should  be.  In  the  pro-active  decision  aid,  a  class  of  recommendation  agents  is  used  to 
provide  explicit  COA  recommendations.  These  agents  are  based  on  the  human-role-playing  and 
the  human-assistant  agents  described  earlier.  While  the  role-playing  agents  are  capable  of 
following  orders  and  reacting  to  the  changing  environment,  the  human-assistant  agents  are  more 
analytical  and  are  able  to  use  trend  and  rule-based  cause-effect  analyses.  Combining  these  agent 
capabilities  with  a  dynamic  alternative  generator  and  utility  function  evaluator  provides  powerful 
real-time  decision  support. 

Essentially,  we  view  the  process  of  decision-making  support  as  general  resource  allocation 
system  (a  system  consisting  of  tasks,  which  require  resources  . . .  sensors,  processors,  humans, 
information,  etc.)  with  functional  and  non-functional  properties.  The  system's  goal  is  to 
recommend  to  the  human  decision  maker,  to  implement  automated  decisions,  and  to  allocate 
resources,  subject  to  the  conflicting  objectives  and  constraints,  in  the  best  possible  way  (there  is 
possibly  significant  competition  for  the  resources).  This  allocation  process  differs  in  level  of 
detail  only,  across  different  stages  of  combat  control  process.  Failures  result  from  poor 
allocation  choices.  Given  the  general  growth  of  the  problem  space,  the  process  has  to  consider 
heuristics  and  approximate  "best  effort"  strategies  (exact  allocation  is  computationally 
prohibitive)  must  be  used.  The  study  provides  a  slate  of  heuristics,  which  can  be  called  explicitly 
by  the  user,  or  by  the  system  under  general  instructions  given  by  the  user.  These  will  include 
commonly  available  and  "home-grown":  genetic,  neural  net,  projection,  greedy,  and  other 
methods  for  designing  and  developing  large-scale,  complex  software.  Three-modi  operandi  will 
be  supported:  search  (entire  space  is  considered  per  iteration),  construction  (a  fixed  number  of 
objects  are  considered  per  iteration),  and  hybrid  (the  heuristic  relies  on  a  strategy/user  to  switch 
modes;  e.g.  first,  do  a  quick  global  search,  then  optimize-construct  a  local  region...). 
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Conclusion 

Eyekon  will  directly  enhance  the  warrior’s  lethality  by  quickly  and  intuitively  locating  targets  in 
the  weapon  sight.  It  will  also  reduce  the  warriors’  vulnerability  by  aiding  in  the  designation  of 
most  dangerous  threat  and  identifying  teammates.  The  result  is  an  AR  system  using  intelligent 
agents  to  superimpose  smart  icons  (Eyekon)  on  the  real  world  weapon  sight  picture.  The  team 
decision  aid  function  of  Eyekon  can  provide  the  team  leader  and  the  team  with  a  method  of 
designating  and  assigning  targets  to  individual  (networked)  soldiers  in  a  very  flexible  robust  self¬ 
organizing  environment. 

The  work  to  date  is  still  in  feasibility  concept  design  to  influence  U.S.  Army  programs.  The  next 
step  is  to  develop  a  prototype  for  evaluation  by  the  U.S.  Army  or  similar  agencies. 
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